Multimodal multi-technique signal fusion system for autonomous vehicle

ABSTRACT

An autonomous vehicle incorporating a multimodal multi-technique signal fusion system is described herein. The signal fusion system is configured to receive at least one sensor signal that is output by at least one sensor system (multimodal), such as at least one image sensor signal from at least one camera. The at least one sensor signal is provided to a plurality of object detector modules of different types (multi-technique), such as an absolute detector module and a relative activation detector module, that generate independent directives based on the at least one sensor signal. The independent directives are fused by a signal fusion module to output a fused directive for controlling the autonomous vehicle.

BACKGROUND

An autonomous vehicle is a motorized vehicle that can operate without ahuman driver. An exemplary autonomous vehicle includes a plurality ofsensor systems, such as, but not limited to, a camera sensor system, alidar sensor system, a radar sensor system, amongst others, wherein theautonomous vehicle operates based upon sensor signals output by thesensor systems. Typically, sensor signals are provided to a computingsystem in communication with the plurality of sensor systems, whereinthe sensor signals capture objects in proximity to the autonomousvehicle, such as traffic lights. The sensor signals are processed by thecomputing system and, based on detection of an object captured in thesensor signal, the processor executes instructions to control amechanical system of the autonomous vehicle (e.g., a vehicle propulsionsystem, a braking system, or a steering system).

Detecting the configuration of an illuminated traffic light can beparticularly suited for camera sensor systems. In such instances, acamera sensor system outputs a sensor signal that defines an image. Anobject detector module incorporated in the computing system detects theconfiguration of one or more illuminated bulbs in one or more trafficlights captured in the image. The detected traffic light configurationis used by the object detector module to define a directive for theautonomous vehicle. The directive may include identifying a permissivemaneuver for the autonomous vehicle to perform based on the detectedconfiguration of the traffic light.

Nevertheless, due to reliance by the computing system upon sensorsignals from a camera sensor system to detect illuminated configurationsof traffic lights, autonomous vehicles that comprise only one camerasensor system are subjected to a single point of failure that couldresult in output of an erroneous directive. For example, if the onlycamera sensor system of the autonomous vehicle is obscured by raindroplets or blocked by dirt, the generated sensor signal may notaccurately capture the configuration of the traffic light. In addition,a computing system having only one object detector module furthersubjects the system to a single point of failure. That is, if the soleobject detector module of the computing system incorrectly identifiesthe configuration of a traffic light, an undesirable directive may beoutput to control the autonomous vehicle.

SUMMARY

The following is a brief summary of subject matter that is described ingreater detail herein. This summary is not intended to be limiting as tothe scope of the claims.

Described herein are various technologies pertaining to a multimodalmulti-technique signal fusion system for an autonomous vehicle. Withmore specificity, described herein are various technologies pertainingto fusing the outputs of a plurality of object detector modules toimprove the suitability of a directive generated for controlling theautonomous vehicle, wherein the outputs are based on at least one sensorsignal such as camera sensor signal. With still more specificity,described herein is a signal fusion system having a fusion module thatgenerates a fused directive by fusing the outputs of the plurality ofobject detector modules, which are each configured to receive the atleast one sensor signal. The outputs of the plurality of object detectormodules are generated to define independent directives based on the atleast one sensor signal provided to each of the plurality of objectdetector modules. A control system controls a mechanical system of theautonomous vehicle, such as a vehicle propulsion system, a brakingsystem, and/or a steering system, based upon the fused directive.

The multimodal multi-technique signal fusion system can include variousaspects that facilitate generation of a fused directive. First, thesystem can be configured to receive multiple sensor signals from aplurality of different sensor systems (i.e., multimodal). Second, thesystem can comprise a plurality of object detection modules includingobject detection modules of different types (i.e., multi-technique).Third, the system fuses the sensor signals to output a fused directiveby merging outputs from the plurality of object detection modules.

In an exemplary embodiment, the plurality of different sensor systemsincludes a fixed exposure camera and an autoexposure camera. The fixedexposure camera may be a general-purpose camera suitable for capturingdaytime images of traffic lights without overexposing the capturedimage. In particular, the fixed exposure camera operates as a desirablesensor system in many instances where ambient light, such as sunlight orstreet lights, brightens the scene to balance the intensity of emittedlight from the captured traffic bulbs. In contrast, capturing anilluminated traffic light at nighttime without sufficient ambient lightmay generate a blurred image that is not readily discernable by theplurality of object detector modules to determine a configuration of thetraffic light. In such cases, an autoexposure camera system, such as ahigh dynamic range (HDR) camera system, may be suitable for capturingnighttime images of traffic lights by maintaining an even exposureacross a captured image on average. For instance, while the majority ofthe scene may be reproduced as dark or blacked out when captured by anHDR camera at nighttime, the illuminated bulbs of the traffic light willbe reproduced in a discernable form without excessive blurring or halosthat hinder the image proximate to the captured traffic light.

It is to be understood from the foregoing that neither fixed exposurecameras or autoexposure cameras are limited to capturing respectivedaytime and nighttime images, nor is any one type of camera specificallyrequired to form a multimodal system. Importantly, it is theimplementation of a signal fusion system that is configured to fuse aplurality of sensor signals (whether from fixed exposure cameras,autoexposure cameras, both, or others) that provides improved accuracyto a directive generated by the signal fusion system.

Incorporated in the signal fusion system is a plurality of objectdetector modules that receive sensor signals generated by the sensorsystems. The plurality of object detector modules includes at least afirst type of module and a second type of module to establish multipletechniques for determining the configuration of a traffic light. Atleast one sensor signal is provided to the plurality of object detectormodules such that each module generates an independentdirective/observation for fusion at the fusion module. In particular,each object detector module is configured to receive each sensor signalgenerated by the plurality of sensor systems. When a plurality of sensorsignals is provided by the plurality of sensor systems, each objectdetector module likewise outputs a plurality of independent directivesthat correspond to a signal from the plurality of the sensor systems(e.g., fixed exposure camera, autoexposure camera, etc.). Thus, whileonly one sensor system is required to perform the multi-technique aspectof the signal fusion system, a plurality of (camera) sensor systems maybe preferable in certain embodiments.

In an exemplary embodiment, the first type of module may be an absolutedetector module and the second type of module may be a relativeactivation detector module. In the context of traffic light detection,an absolute detector is configured to determine a kind of bulb that isilluminated. For example, the absolute detector may use a convolutionneural network to identify a circle that is green in the image andgenerate a directive of “GO”. In contrast, a relative activationdetector is configured to define a directive based on inferences aboutthe traffic light captured in the image. For example, the relativeactivation detector may use a convolution neural network to determinethat a bulb is illuminated in a specific location of a predefinedtraffic light layout and thereby infer, for example, that a green circlehas been detected based on the location of the illuminated bulb in thelayout to generate a directive of “GO”. Accordingly, each of the firsttype of module and the second type of module generate independentdirectives based on a common sensor signal provided to each module.

The independent directives are then provided to a signal fusion moduleto merge the generated outputs of the first type of module and thesecond type of module into a fused directive using a probabilistictechnique. For instance, if outputs from the plurality of the objectdetector modules provide conflicting information, the signal fusionmodule will apply a confidence score to determine which informationshould be incorporated into the fused directive. The fused directivedefines instructions for the autonomous vehicle based on the currentlydetected state of the traffic light as determined by the signal fusionmodule and is provided to a control system of the autonomous vehicle foroperation of a mechanical system thereof.

The above summary presents a simplified summary in order to provide abasic understanding of some aspects of the systems and/or methodsdiscussed herein. This summary is not an extensive overview of thesystems and/or methods discussed herein. It is not intended to identifykey/critical elements or to delineate the scope of such systems and/ormethods. Its sole purpose is to present some concepts in a simplifiedform as a prelude to the more detailed description that is presentedlater.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an exemplary autonomous vehicle.

FIG. 2 illustrates an exemplary architecture that incorporates a signalfusion system.

FIG. 3 illustrates an exemplary architecture that incorporates a signalfusion system.

FIG. 4 is a flow diagram illustrating an exemplary signal fusionprocess.

FIG. 5 is a flow diagram illustrating an exemplary methodology for amultimodal multi-technique signal fusion system.

FIG. 6 is a flow diagram illustrating an exemplary methodology for amultimodal bulb-detector level signal fusion system.

FIG. 7 illustrates an exemplary computing system.

DETAILED DESCRIPTION

Various technologies pertaining to a multimodal multi-technique signalfusion system for an autonomous vehicle is now described with referenceto the drawings, wherein like reference numerals are used to refer tolike elements throughout. In the following description, for purposes ofexplanation, numerous specific details are set forth in order to providea thorough understanding of one or more aspects. It may be evident,however, that such aspect(s) may be practiced without these specificdetails. In other instances, well-known structures and devices are shownin block diagram form in order to facilitate describing one or moreaspects. Further, it is to be understood that functionality that isdescribed as being carried out by certain system components may beperformed by multiple components. Similarly, for instance, a componentmay be configured to perform functionality that is described as beingcarried out by multiple components.

Moreover, the term “or” is intended to mean an inclusive “or” ratherthan an exclusive “or.” That is, unless specified otherwise, or clearfrom the context, the phrase “X employs A or B” is intended to mean anyof the natural inclusive permutations. That is, the phrase “X employs Aor B” is satisfied by any of the following instances: X employs A; Xemploys B; or X employs both A and B.

In addition, the articles “a” and “an” as used in this application andthe appended claims should generally be construed to mean “one or more”unless specified otherwise or clear from the context to be directed to asingular form.

Further, as used herein, the terms “component”, “module”, and “system”are intended to encompass computer-readable data storage that isconfigured with computer-executable instructions that cause certainfunctionality to be performed when executed by a processor. Thecomputer-executable instructions may include a routine, a function, orthe like. It is also to be understood that a component, module, orsystem may be localized on a single device or distributed across severaldevices.

Further, as used herein, the term “exemplary” is intended to meanserving as an illustration or example of something and is not intendedto indicate a preference.

As used herein, the term “fusion” is intended to define the merging of aplurality of outputs into a single output, for example, the merging of aplurality of sensor signals and/or the merging of a plurality ofindependent directives.

As used herein, the term “independent directive” refers to aninstruction that is independently generated by a particular objectdetector from a particular sensor signal to manipulate the movement ofan autonomous vehicle.

As used herein, the terms “directive” and “fused directive” refer to aninstruction that is generated by fusing a plurality of independentdirectives to manipulate the movement of an autonomous vehicle.

With reference now to FIG. 1, an exemplary autonomous vehicle 100 isillustrated. The autonomous vehicle 100 can navigate about roadwayswithout human conduction based upon sensor signals output by sensorsystems of the autonomous vehicle 100. The autonomous vehicle 100includes a plurality of sensor systems 102-104 (a first sensor system102 through an Nth sensor system 104). The sensor systems 102-104 are ofdifferent types and are arranged about the autonomous vehicle 100. Forexample, the first sensor system 102 may be a camera/image sensor systemand the Nth sensor system 104 may be a lidar system; or the sensorsystems 102-104 may be of different types of a particular kind of sensorsystem, such as different types of camera sensor systems. Otherexemplary sensor systems include radar sensor systems, globalpositioning system (GPS) sensor systems, sonar sensor systems, infraredsensor systems, and the like.

A sensor system (e.g., one or more of the plurality of sensor systems102-104) may comprise multiple sensors. For example, the first sensorsystem 102 may comprise a first sensor, a second sensor, etc.Furthermore, some or all of the plurality of sensor systems 102-104 maycomprise articulating sensors. An articulating sensor is a sensor thatmay be oriented (i.e., rotated) by the autonomous vehicle 100 such thata field of view of the articulating sensor may be directed towardsdifferent regions surrounding the autonomous vehicle 100.

The autonomous vehicle 100 further includes several mechanical systemsthat are used to effectuate appropriate motion of the autonomous vehicle100. For instance, the mechanical systems can include but are notlimited to, a vehicle propulsion system 106, a braking system 108, and asteering system 110. The vehicle propulsion system 106 may include anelectric motor, an internal combustion engine, or both. The brakingsystem 108 can include an engine break, brake pads, actuators, and/orany other suitable componentry that is configured to assist indecelerating the autonomous vehicle 100. The steering system 110includes suitable componentry that is configured to control thedirection of movement of the autonomous vehicle 100.

The autonomous vehicle 100 additionally comprises a computing system 112that is in communication with the sensor systems 102-104 and is furtherin communication with the vehicle propulsion system 106, the brakingsystem 108, and the steering system 110. The computing system 112includes a processor 114 and memory 116 that includescomputer-executable instructions that are executed by the processor 114.In an example, the processor 114 can be or include a graphics processingunit (GPU), a plurality of GPUs, a central processing unit (CPU), aplurality of CPUs, an application-specific integrated circuit (ASIC), amicrocontroller, a programmable logic controller (PLC), a fieldprogrammable gate array (FPGA), or the like.

The memory 116 comprises a signal fusion system 118 that is configuredto output a fused directive by merging information generated accordingto a plurality of techniques, wherein the information corresponds to anobject captured in at least one sensor signal provided by at least onesensor system 102-104. The memory 116 additionally includes a controlsystem 120 that is configured to receive the fused directive output bythe signal fusion system 118 and is further configured to control atleast one of the mechanical systems (e.g., the vehicle propulsion system106, the brake system 108, and/or the steering system 110) based uponthe output of the signal fusion system 118.

With reference now to FIG. 2, an architecture 200 is illustrated thatincludes the signal fusion system 118 which outputs a fused directive210 based on at least one sensor signal provided by the sensor systems102-104. The sensor systems 102-104 may include a plurality of camerasensor systems that provides a first sensor signal and a second sensorsignal to the signal fusion system 118. For instance, a first camerasensor system can include a fixed exposure camera and a second camerasensor system can include an autoexposure camera. In an exemplaryembodiment, the sensor systems 102-104 may be configured so that each ofthe plurality of camera sensor systems is adapted to identify aparticular characteristic of an object. For example, a first camerasensor system may provide a first sensor signal that generates grayscale information to capture a shape of an object (e.g., left turningarrow) and a second camera sensor system may provide a second sensorsignal that captures a color of the object (e.g., green). Thus, when thetwo signals are fused, a left-green turning arrow is identified by thesignal fusion module 208 to generate the fused directive 210. Inaddition, the autonomous vehicle 100 may include separate sensor systems102-104 that each capture a specific color of an object (e.g., green,yellow, or red).

A conventional camera sensor system of an autonomous vehicle 100 canhave a viewing range on the order of sixty degrees. However, theincorporation of additional cameras to the autonomous vehicle 100 canincrease the viewing range of the sensor system 102 to one-hundredeighty degrees and beyond, if desirable. The plurality of sensor systems102-104 can include camera sensor systems such as general-purposecameras (fixed exposure) and HDR cameras (autoexposure). Thus, thesignal fusion system 118 is a multimodal system configured to generate adirective based on a plurality of inputs.

The signal fusion system 118 further comprises a plurality of objectdetector modules 202-204 that include at least a first type of moduleand a second type of module to provide multiple object detectiontechniques for a same object captured in at least one sensor signal. Inan exemplary embodiment, the first type of module is an absolutedetector 202 and the second type of module is a relative activationdetector 204. In the context of traffic light detection, an absolutedetector module 202 detects a kind of bulb that is illuminated (e.g.,red circle) to generate a directive for controlling the autonomousvehicle 100. In contrast, the relative activation detector module 204generates a directive for the autonomous vehicle 100 by determining theconfiguration of a traffic light based on inferences about the layout ofthe light. For example, if the top position of a three-bulb verticaltraffic light is illuminated, the relative activation detector 204 mayinfer a “solid red circle” based on predefined layouts incorporated inthe memory 116 to generate an independent directive of “STOP”.

The predefined layouts are selected by the object detector modules202-204 based on a taxonomy that begins at a top level with aconventional traffic light (e.g., a three-bulb, vertically aligned,red-yellow-green light) and branches down through configurations havingincreasing levels of granularity. For example, if the detected trafficlight can be identified more narrowly than the configuration thatcorresponds to the level above it, the object detector modules 202-204continue to distinguish the traffic light at further levels ofgranularity in the taxonomy, such as by differentiating betweenred-yellow-green-green arrow traffic lights and red-yellow-green-redarrow traffic lights. This process is conducted via a convolution neuralnetwork until an illuminated configuration of the traffic light ispaired with the most granular predefined layout that it can be matchedto in a database of layouts.

Each type of object detector module (e.g., absolute detector module 202and relative activation detector module 204) generates an independentdirective for each sensor signal provided by the sensor systems 102-104;the absolute detector module 202 generates independent directives 206 aand the relative activation detector module 204 generates independentdirectives 206 b (the independent directives 206 a and the independentdirectives 206 b are collectively referred to herein as independentdirectives 206). Each of the independent directives 206 define a(pre-fusion) vehicle maneuver based on the state of illuminationdetected by the object detector modules 202-204 according to thepredefined layouts. The independent directives 206 may be fused at theobject detector/bulb level by the plurality of object detector modules202-204 when a same object detector module generates a same independentdirective 206 for a same sensor signal. Otherwise, the plurality ofindependent directives 206 are provided to the signal fusion module 208where the independent directives 206 are thereby merged/fused.

Each independent directive 206 provided to the signal fusion module 208defines a vehicle maneuver that corresponds to the observed state of thetraffic light. The fusion module 208 then applies confidence scores tothe observations captured in the sensor signal(s) to determine theaccuracy of the detected traffic light layout and illuminatedconfiguration thereof. For instance, a first independent directive maycorrespond to a solid red circle, whereas a second independent directivemay correspond to a flashing red circle. The signal fusion module 208fuses the first and second independent directives to output a fuseddirective 210 that defines a vehicle maneuver based on the illuminatedstate of the traffic light, as determined by the signal fusion module208 according to confidence scores applied to the independent directivesthat were based on identification of a solid red circle and a flashingred circle.

Referring now to FIG. 3, an architecture 300 incorporating the signalfusion system 118 is illustrated. The architecture 300 includes atraffic light copilot 304 and a region of interest module 306 disposedin signal communication between the sensor systems 102-104 and thesignal fusion system 118. In other embodiments the region of interestmodule 306 may be incorporated within the traffic light copilot 304.Accordingly, the architecture 300 includes nodes at the traffic lightcopilot 304, the object detector modules 302, and the signal fusionmodule 208. The traffic light copilot 304 defines a geometric projectionthat identifies where an object, such as a traffic light, is positionedrelative to the sensor systems 102-104 of the autonomous vehicle 100.The output of the traffic light copilot 304 is provided to the region ofinterest module 306 to define a region of interest around a light sourcecaptured in the sensor signal of an image sensor system. In an exemplaryembodiment, the region of interest comprises dimensions that areconsiderably larger than a traffic light (e.g., the region of interestmay correspond to a height of 3 meters when defined around a trafficlight having a height of 1 meter).

Additionally included in the architecture 300 is a convolution neuralnetwork 308 and a directive state machine 310. The convolution neuralnetwork 308 is linked to the object detector modules 302 to identifyobjects/configurations in the region of interest that is defined by theregion of interest module 306. In an exemplary embodiment, a pluralityof convolutional neural networks 308 can be running on a same imagesensor signal to detect a plurality of objects/configurations capturedin the sensor signal.

The directive state machine 310 is in communication with the signalfusion module 208 and is configured to define at least eight universaldirectives including: STOP (red light), STOP_AND_YIELD (flashing redlight), MAYBE_STOP (yellow light), YIELD (flashing yellow light),ABOUT_TO_GO (light will soon turn green—transition directive in somecountries), GO (green light), GO_PROTECTED (proceed through), andUNKNOWN (no detected light). A directive defines the most suitablecourse of action that an autonomous vehicle 100 should perform accordingto the configuration of the traffic light/lane and the applicable lawsof the region. For instance, it is permissible for an autonomous vehicle100 to exit an intersection on a solid red light (GO_PROTECTED) but itis not permissible for the autonomous vehicle 100 to enter theintersection on a solid red light without stopping. As such, the lattercircumstance would be in contrast with the former circumstance, whereinthe latter circumstance corresponds to a directive of STOP or, in statesthat allow vehicles to make a right-on-red, a directive ofSTOP_AND_YIELD.

Referring now to FIG. 4, a flow diagram 400 of an exemplary signalfusion process is illustrated. The flow diagram 400 includes sensorsignals/images 402-404 generated by a first camera and a second camerathat capture a solid red light 412 disposed next to an alternatingflashing red light 410, such as a combination of traffic lights that maybe found at a railroad crossing. The first image 402 corresponding tothe first camera may be generated by a fixed exposure camera and thesecond image 404 corresponding to the second camera may be generated byan autoexposure camera. Each set of images 402-404 captures the pair oftraffic lights 410-412 and is processed by executing instructions onobject detector modules of different types (e.g., an absolute detectormodule and a relative activation detector module) to generate eightobservations 406 that correspond to eight independent directives (i.e.,two observations per image, times two images, times two types of objectdetector modules).

In the exemplary images 402-404, a traffic light copilot 304 detectsthat the same two light emitting sources are captured in a sensor signalof the first camera and a sensor signal of the second camera.Accordingly, the traffic light copilot 304 generates correspondingsignals for the region of interest module 306 to define regions ofinterest around each of the two light emitting sources captured in thesensor signals. The regions of interest are configured to circumscribeeach of the light emitting sources as oversized boxes in comparison tothe expected size of a traffic light, so that if the light emittingsource is determined to correspond to a traffic light, the traffic lightwill be fully confined within the region of interest. That is, if theregions of interest were configured to be the same size as aconventional traffic light, it is possible that some bulbs of thetraffic light would fall outside the region of interest when the regionof interest module 306 centralized the light emitting source within aregion of interest box. This would be especially apparent when the lightemitting source is associated with less common traffic light layouts andconfigurations.

The images 402-404 are processed by a plurality of object detectormodules via a convolution neural network 308 that identifiesconfigurations of the light emitting sources in the regions of interest.In the exemplary images 402-404, a solid red light 412 and analternating flashing red light 410 are detected by the convolutionneural network 308, which provides corresponding signals to the objectdetector modules. The object detector modules generate an independentdirective for each traffic light captured in each image provided to eachobject detector module, thereby accumulating eight observations 406 thatform the basis of signal fusion 408.

If the detected traffic signals are correctly determined by the objectdetector modules, four of the independent directives would correspond toan alternating flashing red light 410 (STOP_AND_YIELD) and four of theindependent directives would correspond to a solid red light 412 (STOP).If one of the cameras or object detector modules generates a signal thatincorrectly identifies one of the lights 410-412, a third type ofindependent directive would be generated. All of the independentdirectives are merged by signal fusion 408 using probabilistictechniques based on confidence scores. In the embodiment describedabove, merging four STOP_AND_YIELD directives with four STOP directiveswould result in a fused directive of STOP, which is output to thecontrol system of the autonomous vehicle for manipulating operationthereof.

FIGS. 5 and 6 illustrate exemplary methodologies relating to amultimodal multi-technique signal fusion system for an autonomousvehicle. While the methodologies are shown and described as being aseries of acts that are performed in a sequence, it is to be understoodand appreciated that the methodologies are not limited by the order ofthe sequence. For example, some acts can occur in a different order thanwhat is described herein. In addition, an act can occur concurrentlywith another act. Further, in some instances, not all acts may berequired to implement a methodology described herein.

Moreover, the acts described herein may be computer-executableinstructions that can be implemented by one or more processors and/orstored on a computer-readable medium or media. The computer-executableinstructions can include a routine, a sub-routine, programs, a thread ofexecution, and/or the like. Still further, results of acts of themethodologies can be stored in a computer-readable medium, displayed ona display device, and/or the like.

Referring now to FIG. 5, an exemplary methodology 500 for a multimodalmulti-technique signal fusion system for an autonomous vehicle isillustrated. The methodology 500 starts at 502, and at 504 at least onesensor signal is generated by at least one sensor system. The at leastone sensor signal captures the configuration of an object, such as anilluminated state of a traffic light. The at least one sensor system maybe an image sensor system, an infrared sensor system, or any other typeof sensor system that is suitable for detecting an illuminated bulb of atraffic light.

At 506, the at least one sensor signal is provided to a signal fusionsystem, wherein the signal fusion system includes a plurality of objectdetector modules in communication with a signal fusion module. At 508,each of the plurality of object detector modules receive the at leastone sensor signal and thereby generate independent directives based onthe at least one sensor signal, wherein the independent directivesdefine a traffic maneuver to be performed by the autonomous vehicle. At510, the signal fusion module fuses the independent directives to outputa fused directive. The independent directives are fused into a fuseddirective according to a probabilistic technique that assigns confidencescores to each of the independent directives. The fused directive isprovided to a control system and defines instructions to be executed forcontrolling the autonomous vehicle. At 512, the control system of theautonomous vehicle controls a mechanical system, such as a vehiclepropulsion system, a braking system, and/or a steering system, based onthe fused directive. The methodology completes at 514.

Referring now to FIG. 6, an exemplary methodology 600 for a multimodalbulb-detector level signal fusion system is illustrated. The methodology600 starts at 602, and at 604 sensor signals are generated by aplurality of sensor systems. The sensor signals capture theconfiguration of an object, such as an illuminated state of a trafficlight. The plurality of sensor systems may include one or more imagesensor systems, infrared sensor systems, or any other type of sensorsystems that are suitable for detecting an illuminated bulb of a trafficlight.

At 606, each of the sensor signals are provided to an object detectormodule in communication with each of the plurality of sensor systems. At608, independent directives are generated by the object detector modulethat correspond to the sensor signals, wherein the independentdirectives define a maneuver to be performed by the autonomous vehicle.At 610, the object detector module fuses the independent directives tooutput a fused directive. The independent directives are fused into afused directive according to a probabilistic technique by the objectdetector module. The fused directive is provided to a control system anddefines instructions to be executed for controlling the autonomousvehicle. At 612, the control system of the autonomous vehicle controls amechanical system, such as a vehicle propulsion system, a brakingsystem, and/or a steering system, based on the fused directive. Themethodology completes at 614.

Referring now to FIG. 7, a high-level illustration of an exemplarycomputing device 700 that can be used in accordance with the systems andmethodologies disclosed herein is illustrated. For instance, thecomputing device 700 may be or include the computing system 112. Thecomputing device 700 includes at least one processor 702 that executesinstructions that are stored in a memory 704. The instructions may be,for instance, instructions for implementing functionality described asbeing carried out by one or more modules and systems discussed above orinstructions for implementing one or more of the methods describedabove. In addition to storing executable instructions, the memory 704may also store location information, distance information, directioninformation, etc.

The computing device 700 additionally includes a data store 708 that isaccessible by the processor 702 by way of the system bus 706. The datastore 708 may include executable instructions, location information,distance information, direction information, etc. The computing device700 also includes an input interface 710 that allows external devices tocommunicate with the computing device 700. For instance, the inputinterface 710 may be used to receive instructions from an externalcomputer device, etc. The computing device 700 also includes an outputinterface 712 that interfaces the computing device 700 with one or moreexternal devices. For example, the computing device 700 may transmitcontrol signals to the vehicle propulsion system 106, the braking system108, and/or the steering system 110 by way of the output interface 712.

Additionally, while illustrated as a single system, it is to beunderstood that the computing device 700 may be a distributed system.Thus, for instance, several devices may be in communication by way of anetwork connection and may collectively perform tasks described as beingperformed by the computing device 700.

Various functions described herein can be implemented in hardware,software, or any combination thereof. If implemented in software, thefunctions can be stored on or transmitted over as one or moreinstructions or code on a computer-readable medium. Computer-readablemedia includes computer-readable storage media. A computer-readablestorage media can be any available storage media that can be accessed bya computer. By way of example, and not limitation, suchcomputer-readable storage media can comprise RAM, ROM, EEPROM, CD-ROM orother optical disk storage, magnetic disk storage or other magneticstorage devices, or any other medium that can be used to store desiredprogram code in the form of instructions or data structures and that canbe accessed by a computer. Disk and disc, as used herein, includecompact disc (CD), laser disc, optical disc, digital versatile disc(DVD), floppy disk, and Blu-ray disc (BD), where disks usually reproducedata magnetically and discs usually reproduce data optically withlasers. Further, a propagated signal is not included within the scope ofcomputer-readable storage media. Computer-readable media also includescommunication media including any medium that facilitates transfer of acomputer program from one place to another. A connection, for instance,can be a communication medium. For example, if the software istransmitted from a website, server, or other remote source using acoaxial cable, fiber optic cable, twisted pair, digital subscriber line(DSL), or wireless technologies such as infrared, radio, and microwave,then the coaxial cable, fiber optic cable, twisted pair, DSL, orwireless technologies such as infrared, radio and microwave are includedin the definition of communication medium. Combinations of the aboveshould also be included within the scope of computer-readable media.

Alternatively, or in addition, the functionally described herein can beperformed, at least in part, by one or more hardware logic components.For example, and without limitation, illustrative types of hardwarelogic components that can be used include Field-programmable Gate Arrays(FPGAs), Program-specific Integrated Circuits (ASICs), Program-specificStandard Products (ASSPs), System-on-a-chip systems (SOCs), ComplexProgrammable Logic Devices (CPLDs), etc.

What has been described above includes examples of one or moreembodiments. It is, of course, not possible to describe everyconceivable modification and alteration of the above devices ormethodologies for purposes of describing the aforementioned aspects, butone of ordinary skill in the art can recognize that many furthermodifications and permutations of various aspects are possible.Accordingly, the described aspects are intended to embrace all suchalterations, modifications, and variations that fall within the spiritand scope of the appended claims. Furthermore, to the extent that theterm “includes” is used in either the detailed description or theclaims, such term is intended to be inclusive in a manner similar to theterm “comprising” as “comprising” is interpreted when employed as atransitional word in a claim.

What is claimed is:
 1. An autonomous vehicle, comprising: a mechanicalsystem; at least one sensor system, comprising: a fixed exposure camerasystem configured to output a first sensor signal; and an auto exposurecamera system configured to output a second sensor signal; a computingsystem in communication with the mechanical system and the at least onesensor system, wherein the computing system comprises: a processor; andmemory that stores instructions that, when executed by the processor,cause the processor to perform acts comprising: generating a firstindependent directive and a second independent directive that eachdefine a vehicle maneuver based upon an illuminated configuration of atraffic light captured in the first sensor signal and the second sensorsignal; fusing the first independent directive and the secondindependent directive to output a fused directive; and controlling themechanical system based upon the fused directive.
 2. The autonomousvehicle of claim 1, wherein the first independent directive and thesecond independent directive are generated by a plurality of objectdetectors that receive the first sensor signal and the second sensorsignal from the at least one sensor system.
 3. The autonomous vehicle ofclaim 2, wherein one of the first sensor signal or the second sensorsignal captures a shape of an object and the other one of the firstsensor signal or the second sensor signal captures a color of theobject.
 4. The autonomous vehicle of claim 2, wherein the plurality ofobject detectors includes an absolute detector to generate the firstindependent directive and a relative activation detector to generate thesecond independent directive.
 5. The autonomous vehicle of claim 2,wherein the first independent directive and the second independentdirective are fused by the plurality of object detectors when the firstindependent directive and the second independent directive are a samedirective output by a same type of object detector.
 6. The autonomousvehicle of claim 1, wherein the memory further stores instructions, thatwhen executed by the processor, cause the processor to perform actscomprising: determining, utilizing a convolution neural network, theilluminated configuration of the traffic light, the illuminatedconfiguration of the traffic light being one of a number of predefinedtraffic light layout configurations.
 7. The autonomous vehicle of claim1, wherein the first independent directive and the second independentdirective are fused to generate the fused directive based upon aconfidence score.
 8. The autonomous vehicle of claim 7, wherein thefused directive defines a post-fusion vehicle maneuver based uponconfirmation of the illuminated configuration of the traffic light.
 9. Amethod performed by an autonomous vehicle, the method comprising:generating a first sensor signal and a second sensor signal, wherein thefirst sensor signal is generated by a fixed exposure camera system ofthe autonomous vehicle and the second sensor signal is generated by anauto exposure camera system of the autonomous vehicle; generating afirst independent directive and a second independent directive that eachdefine a vehicle maneuver based upon an illuminated configuration of atraffic light captured in the first sensor signal and the second sensorsignal; fusing the first independent directive and the secondindependent directive to output a fused directive; and controlling amechanical system of the autonomous vehicle based upon the fuseddirective.
 10. The method of claim 9, further comprising generating thefirst independent directive and the second independent directive by aplurality of object detectors that receive the first sensor signal andthe second sensor signal from the fixed exposure camera system and theauto exposure camera system.
 11. The method of claim 10, furthercomprising fusing the first independent directive and the secondindependent directive when the first independent directive and thesecond independent directive are a same directive output by a same typeof object detector.
 12. The method of claim 9, further comprisinggenerating the first independent directive by an absolute detector andthe second independent directive by a relative activation detector. 13.The method of claim 9, further comprising determining the illuminatedconfiguration of the traffic light utilizing a convolution neuralnetwork, the illuminated configuration of the traffic light being one ofa number of predefined traffic light layout configurations.
 14. Themethod of claim 9, further comprising fusing the first independentdirective and the second directive to generate the fused directive basedupon a confidence score.
 15. An autonomous vehicle, comprising: amechanical system; at least one sensor system configured to output afirst sensor signal and a second sensor signal, wherein the first sensorsignal captures a shape of an object and a second sensor signal capturesa color of the object; a computing system in communication with themechanical system and the at least one sensor system, wherein thecomputing system comprises: a processor; and memory that storesinstructions that, when executed by the processor, cause the processorto perform acts comprising: generating a first independent directive anda second independent directive that each define a vehicle maneuver basedupon an illuminated configuration of a traffic light captured in thefirst sensor signal and the second sensor signal; fusing the firstindependent directive and the second independent directive to output afused directive; and controlling the mechanical system based upon thefused directive.
 16. The autonomous vehicle of claim 15, wherein thefirst independent directive and the second independent directive aregenerated by a plurality of object detectors that receive the firstsensor signal and the second sensor signal from the at least one sensorsystem.
 17. The autonomous vehicle of claim 16, wherein the plurality ofobject detectors includes an absolute detector to generate the firstindependent directive and a relative activation detector to generate thesecond independent directive.
 18. The autonomous vehicle of claim 15,wherein a first camera system outputs the first sensor signal and asecond camera system outputs the second sensor signal.
 19. Theautonomous vehicle of claim 15, wherein the at least one sensor systemcomprises a fixed exposure camera and an auto exposure camera.
 20. Theautonomous vehicle of claim 15, wherein the memory further storesinstructions, that when executed by the processor, cause the processorto perform acts comprising: determining, utilizing a convolution neuralnetwork, the illuminated configuration of the traffic light, theilluminated configuration of the traffic light being one of a number ofpredefined traffic light layout configurations.